Preferential Physics - pilot analysis and preregistration!

This document is the preregistration for the Preferential Physics project on Lookit. Data collection is currently underway, but sample size has been set prior to data collection kickoff (50 participants reaching 12th session), and the vast majority of videos haven’t even been coded for looks, much less critical DVs calculated.

This document contains a brief orientation to the project, links to relevant other documentation, and a reproducible set of analyses that reads in a set of pilot data, generates a simulated dataset, and attempts to set up the model comparisons we actually care about!

Introduction

The idea is to use dense sampling of individual infants on Lookit to conduct a detailed assessment of understanding of several physical principles. How stable are individual components across sessions and how independent? What does partial knowledge look like at the individual level?

We are interested in infants’ preferential looking ratio to simple violations of:

• Gravity: Completely unsupported objects should fall down immediately, rather than moving up, continuing in their current trajectory, or moving down at some delay.

• Inertia: Objects should continue roughly in their current trajectory when gravity is not a factor, rather than stopping and starting or turning around.

• Support: which of the following should fall (vs. stay put) after being placed?

  • An object placed mostly on the anchor

  • An object placed only slightly on the anchor

  • An object touching the side or bottom of the anchor

  • An object near the anchor but not touching it

Big picture

Why study individual behavior in more detail–what do we lose in studying groups of kids? We don’t know the extent to which a success means that a significant fraction of kids this age can do this task VS that all kids of this age can do this task to a significant degree. And even looking at the distribution of scores doesn’t clear this up, unless it’s an incredibly strong result (e.g. all kids get 9-10 of 10 questions right, or no kids get more than 6 out of 10). This matters for

• understanding how abilities are related to each other: we can’t get nearly as much out of age-based progressions without knowing how the noise works—especially for results of the form “n-month-olds, but not m-month-olds, can do X”

• understanding what partial knowledge & mechanisms of change in a domain look like: when kids “fail,” is that some kids making a correct prediction and some making an incorrect prediction? Or are they all failing to make any prediction and/or predicting at chance? When kids succeed but not at ceiling, are some getting one aspect and some another?

It may be that some kids don’t express nearly-universal knowledge on the dependent measures we collect. We can evaluate this explanation for noise and/or individual differences by studying the methods themselves, and kids’ behavior on them: e.g., can we predict the types of preferential looking responses we get from kids based on control tasks? How stable are those controls and task performance across sessions? (Especially interesting would be differences within kids in expression: kids may genuinely express knowledge at some times, but not others, due to attentional/emotional state changes.)

Especially in development, where we’re interested in the underpinnings of human cognition, the difference between “some babies use this type of information but others do something else” and “all babies have this type of information available to them, unless something’s wrong” matters a lot—this is exactly where we care about universality.

Experimental paradigm

Children complete 24 20-second preferential looking trials per session; families are encouraged to complete 15 sessions within 2 months. Parents complete a short mood survey and go through some instructions before the preferential looking portion. They’re asked to hold their children looking over their shoulders during this portion to avoid parental bias. Parents can end the study at any point and skip to the post-study survey.

Each trial begins with an object intro (video of Kim saying “Look, this is a …”) and demonstrating use of an object – e.g. biting into an apple, putting on hand lotion, drawing with a marker, eating with a spoon) that lasts about 5 seconds.

This is intended as an attention-getter to re-orient children towards the center, while in principle reinforcing that the object in question is not an agent and should be expected to follow normal physical laws.

Then, two events involving that object are shown simultaneously, one on the left and one on the right, looping for 24s (event videos range from about 2-5s). Events always show the same object, same camera angle, same background, with a difference only in the “outcome.” Event types are shown in the table below; each concept is presented 4-6 times, with 2-3 repetitions of each event type. Although events are short, they loop continuously for 20s. Realtime events are shown so that “expected” events are at natural speeds, and not potentially seen as violating physical principles due to happening too slowly.

Parents can pause individual trials. If they pause during the intro, they just start over upon restarting. If they pause during the test (up to one time per trial) they restart from the intro, but then the left and right test videos are switched for the test phase.

Trial assignment & ordering

(See below for descriptions of these trial types.) Trials cycle through gravity, inertia+calibration, support, and control (same/salience) pairings during a session; the order of these concepts is chosen from a list and changed (cycling through a list of orders) each session. There are six videos shown in each category in total.

Within each category, objects are assigned to comparison types (e.g. “apple” assigned to “table, down vs. up”) by choosing from a list of acceptable mappings, again incremented per session. (The first session value was selected randomly from the first six options initially [by accident], and is now selected from all possible mappings.)

There are six possible comparisons for the stay and fall events; three comparisons are assigned to stay and three to fall, with the selection again cycling through a random list of such assignments per session. Left/right placement, horizontal flipping of the left and right events, camera angles, and backgrounds are chosen randomly with the constraint that half of the ‘more probable’ events are on the left within each category. Calibration trials (grouped with inertia videos for purposes of assigning object intros) are placed at trials 3 and 6, so that they are always available for kids who completed enough trials for the session to be included (and so that if there are differences in coding quality across trials, we’re not excluding on the basis of when calibration happened).

Event types

Gravity

Table

Object is rolled/slid off a table and continues down, horizontal, or up.

Toss

Object in hand is tossed down, falls UP, tossed up, falls DOWN.

Ramp

Object is placed in center of ramp and released to roll down or up.

Inertia

Stop

Object rolls from one side of screen and stops in the middle, then re-starts on its own, or by a hand.

Reverse

Object rolls/slides from one side of the screen and collides with a barrier, or takes the same trajectory colliding against no bairrier.

Support

Fall

An object is placed (mostly on/slightly on/next to/near) on a cabinet and immediately falls.

Stay

An object is placed (mostly on/slightly on/next to/near) on a cabinet and stays there.

Control

Same

Distinguishable but similar physically-possible human actions on objects, like rotating an object about one axis vs. another

Salience

Physically-possible human actions on objects, some more interesting, like flipping a spoon vs. slowly extending it or erasing a drawing vs. an empty board.

Calibration

A spinning ball moves across the entire screen.

Dataset contents

Here we describe the information that will be available in the final dataset. Some of this info is not available for the pilot dataset, in particular anything having to do with more than one session.

Per child

• Age at start

• Demographic info (optional, typically reported): family income, languages spoken at home, parent education level(s), number of parents in the home, number of children’s books in the home, child’s race, number and age of siblings, country + US state if in US, urban/suburban/rural

Per session

• Child’s age

• Number of previous sessions completed

• Time since last session

• Mood data (Before beginning study, by parent report. Scales 1-7: CHILD: tired-rested, sick-healthy, fussy-happy, calm-active; PARENT: tired-energetic, overwhelmed-ontopofthings, upset-happy. How long since child woke up, how long since child ate, how long until child is due for nap/sleep; what child was doing before this.)

Per trial

• Time looking L, R (full time sequence of looks left, right, away)

• Number of fixations (derived from the first one)

• Proportion looking to L, out of time looking to screen (derived from the first one)

• Parent behavior: times of talking, pointing, and peeking

• Infant behavior: times of fussing & rating of fussiness level low or high

• Trial number

Predictors for fraction time to left

• Comparison type (e.g. ramp up vs. down), nested within

• Event type (e.g. ramp), nested within…

• Concept (gravity, inertia, support)

• Object (apple, lotion bottle, scissors…)

• Whether each side is unexpected (i.e., does the event on the left clearly violate a physical principle? Does the event on the right? Sometimes both are unexpected to adults, e.g. when an object near a cabinet and an object next-to a cabinet stay put; sometimes neither is. In some cases this depends on the child’s potential beliefs, see modeling…) & which side is less expected.

Data acquisition and exclusion

Sample size & recruitment procedures

Age range 4-12 months at start of study; continue for up to 61 days; target ‘complete’ dataset is 12 usable sessions. (No major age differences in data quality or salience/same controls seen in piloting.)

Plan to recruit as large a balanced sample as practical given time constraints on both testing and recruiting; aiming for 50 participants with a complete dataset. All recruitment decisions are / have been made without examining dependent variables. This will also be the case if we must terminate data collection early (i.e. for timing/funding constraints)

Data exclusion & disambiguation

All partial datasets (<12 sessions) and any extra data collected will be included in the analysis. Data will only be excluded from analysis if it meets any of the criteria below; we aim to include as much data as possible and use analyses that are robust to missing/‘unbalanced’ data. Data may be excluded at the level of the participant (i.e. all data from that child is excluded), sessions, or individual trial.

Note also that because (a) we may exclude some-but-not-all data from a participant and (b) the number of times baby is taken to have seen the study is relevant, the ‘study number’ (= number of sessions seen) may differ from a simple count of the sessions that are present in the dataset.

Child level

• Gestational age at birth < 37 weeks, for any analyses using age. Unknown gestational age will be used but prevalence reported. (Followup to check that inclusion of premature infants does not qualitatively affect other results, and/or to display results from premature infants with adjusted/non-adjusted age - exploratory.)

• Children who participated in the pilot study

• Children whose parents spontaneously report developmental/medical issues that would likely explain some differences in task: vision or hearing impairment; cognitive or neurological disorders including due to trisomies.

• Note again: We include data from children with any number of sessions (will probably have many with <12 in addition to “complete” data); analyses described should be able to handle this appropriately.

Session level

• Sessions where children are outside age range of 4-14 months, except for binned-by-age analyses where we may display data from children outside the age range (without affecting any other values) if we end up having it. Adjusted age will be used for premature infants.

• Sessions with < 6 trials (& don’t count as a session for session # purposes). Parents are encouraged not to complete sessions within 6 hours of each other. For each session (process later -> earlier), if another session with fewer completed trials happened within six hours, use this one instead.This leads to reasonable outcomes even in the unlikely event someone’s doing the study every 5 hours.

• Require calibration performance >75% to use session. Calibration scores seem to be mostly due to difficulty coding and might therefore index overall confidence in other judgments; timing differences in webcam stream vs. displayed stimuli will also affect calibration. Pool all looking across the two calibration trials to compute an overall calibration score, so that if kids aren’t looking as much for one of the trials we don’t average in a much noisier measurement. Score is fractional looking time to correct side during the middle of periods when the ball should be static: [0.5, 3.5], [5.5, 8.5], [10.5, 13.5], [15.5, 18.5], [20.5, 23.5].

• If the participant has <12 usable sessions spread over >60 days, use sessions from the earliest 60-day period (inclusive) with the most usable sessions. If the participant has >=12 usable sessions over >60 days, use sessions from the earliest 60-day period (inclusive) with at least 12 sessions.

• Where absolute session number (as an index of how many times the child has seen stimuli, etc.) is relevant, assign the first session used in analysis a session number according to the number of ‘experienced sessions’ in the preceding 60 days. Experienced sessions include sessions where the child is out of age range or calibration performance is poor, but not sessions with <6 trials. (For instance, if a child participates at 4 months of age and then 12 times from 10-11 months, the latter set of data is used and the first session at 10 months is considered session 1. If a child participates on days 1, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, then sessions starting at day 30 are used but the first session number is 2.

Trial level

• Require >= 2s looking to use a trial. Don’t otherwise deal with shorter/longer LTs except in (exploratory) model.

• Omit periods where the child is out of frame or gaze is otherwise impossible to code. Treat as out of frame any periods where the video is ‘frozen’ for >1s (start treating as out of frame 1s after this period begins) and report prevalence.

• Exclude trials where child is fussing >50% of the trial. In contrast to omitting periods where child is fussing, this avoids dependence on exactly which frames are considered fussy, at the cost of a threshold effect we expect not to affect many trials. >50% applies to length of video, not otherwise codable data (i.e. we allow fuss coding during outofframe periods).

• Parent interference: Exclude periods of trials after parent peeks, points while peeking, or speaks in any way that could bias child (“what’s that ball doing?” but not “keep looking sweetie!”) Include periods where the parent’s eyes are not visible and we can’t tell where they’re looking (unless there is reason to believe they are looking) and periods where the parent is looking away but may see stimuli peripherally.

Pilot data, visualizations, and sanity checking

We read in the pilot datafile produced by Kim [TODO: Insert dates of test], but for replicability, need to produce a dataset that definitely doesn’t include PII that can be shared alongside the analysis.

We’ll begin by visualizing the pilot dataset for some basic properties. First, we expect that babies’ total looking time will drop over time.

Next, we’ll plot the critical DV - how much time do babies spend looking at surprising things? Note a complication of the dataset here: we will be modeling the ‘raw’ dependent variable, fracLeft (out of some amount of time staring at the screen, what percentage of that time is spent looking at the left-hand video?) and nd asking whether this proportion is affected by lessExpectedLeft - is whichever video is most surprising (from an adult point of view) located on the left or the right?

Remember that lessExpectedLeft corresponds to a different pairing of video types for every event and concept. Critically, this value is undefined for some event types, in particular the SAME events (TODO: add a link to above.)

For ease of visualization, we’ll be plotting fracLessExpected: percentage of time spent looking at which ever video is most surprising. Here is a box plot of looks to these less expected things by concept.

And the same, organized by age.

Build the basic model

Analyses and Preregistration (Simulated dataset)

Discussion/summary of preregistered hypotheses

Construct ‘dummy’ data with some basic expected characteristics

Create simulated data for 12 sessions! We do this by just duplicating the dataset with increasing (equal) slope toward fracLessExpected = 1 (full attention toward most surprising, from adult perspective), and increasing ages by a few days at a time. For items where lessExpected is undefined, we add a bit of noise but leave alone.

#Key columns
# shortId - participant identifier (uuid)
# proquint - readable ID, in case we want to examine cases
# ageRegistration - child age in months + fractional months
# stimuli - stimulus identifier
# trialnum - trial number
# comparison - specific comparison in physics concept (e.g. "on vs. near")
# event - physics event (ramp, etc)
# concept - physics event *category* (lumps a few events together)
# lessExpectedLeft - location of the less expected version (undefined for items that don't have a less expected version!)
# fracLessExpected - Fractional looking time to those trials (undefined for items that don't have a less expected version)
# totalLT - Total trial time in case we want to recover that!

# Construct simulated data (Note, we can make this more complicated over time)
# For now, it assumes that fracLessExpected values just get higher with sessions, on slopes
# determined semi randomly for each child/concept, and that ages get older on a slighly random schedule

simulated_12session_data <- pilotdata %>%
  filter(!is.na(fracLessExpected))%>%
  group_by(shortId, concept) %>%
  dplyr::summarize(startMean_sim = mean(fracLessExpected), slope_sim = abs(rnorm(1,0.05,0.01)), startAge = mean(ageRegistration)) %>%#Small, usually positive slopes
  uncount(12, .id = 'session') %>%
  left_join(filter(pilotdata, !is.na(fracLessExpected))) %>%
  mutate(sim_fracLessExpected = startMean_sim + (session-1)*slope_sim + rnorm(1,0, 0.01)) %>%
  mutate(sim_fracLessExpected = ifelse(sim_fracLessExpected < 1, sim_fracLessExpected, 1)) %>%
  mutate(sim_age = startAge + (session-1)*abs(rnorm(1,0.25,0.5))) #Average 1 week between sessions, no allow negative values

simulated_12session_data <- pilotdata %>%
  filter(is.na(fracLessExpected))%>%
  group_by(shortId, concept) %>%
  dplyr::summarize(startMean_sim = mean(fracLessExpected), slope_sim = abs(rnorm(1,0.05,0.01)), startAge = mean(ageRegistration)) %>%#Small, usually positive slopes
  uncount(12, .id = 'session') %>%
  left_join(filter(pilotdata, !is.na(fracLessExpected))) %>%
  mutate(sim_fracLessExpected = startMean_sim + (session-1)*slope_sim + rnorm(1,0, 0.01)) %>%
  mutate(sim_fracLessExpected = ifelse(sim_fracLessExpected < 1, sim_fracLessExpected, 1)) %>%
  mutate(sim_age = startAge + (session-1)*abs(rnorm(1,0.25,0.5))) #Average 1 week between sessions, no allow negative values

Initial visualizations - SIMULATED DATA

• And again, with the simulated data

concept_means_per_child_per_session <- simulated_12session_data %>%
  filter(!is.na(fracLessExpected))%>%
  group_by(shortId, concept, session) %>%
  summarise(fLE = mean(sim_fracLessExpected), age = first(sim_age))

g <- ggplot(concept_means_per_child_per_session, aes(y = fLE, x = concept)) +
  geom_boxplot() +
  #facet_wrap(~session) + 
  #geom_point(alpha = 0.05) +
  geom_jitter(alpha = 0.1) +
  geom_hline(yintercept=0.5, linetype = 'dashed')+
  ggtitle('SIMULATED - grand means per concept')
g

• Graphing by age, each event type separately. (This looks silly because the data is simulated - age and attention to fLE for all concepts both set to increase stepwise)

concept_means_per_child_per_session <- simulated_12session_data %>%
  group_by(shortId, session, concept) %>%
  summarise(fLE = mean(sim_fracLessExpected), age = first(sim_age))

g <- ggplot(concept_means_per_child_per_session, aes(y = fLE, x = age)) +
  geom_jitter(alpha = 0.1) +
  geom_smooth(method = "lm")+
  geom_hline(yintercept=0.5, linetype = 'dashed')+
  facet_wrap(~concept) + 
  ggtitle('Means by age')
g

• Graphing within concepts, e.g. ‘stay’

stay_means_per_child_per_comparison <- simulated_12session_data %>%
  filter(event == 'stay') %>%
  group_by(shortId, session, comparison) %>%
  summarise(fLE = mean(sim_fracLessExpected), age = first(sim_age))

g <- ggplot(stay_means_per_child_per_comparison, aes(y = fLE, x = age)) +
  geom_jitter(alpha = 0.1) +
  geom_smooth(method = "lm")+
  geom_hline(yintercept=0.5, linetype = 'dashed')+
  facet_wrap(~comparison) + 
  ggtitle('Stay')
g

Then, build up the models

An effect of trial number on TOTAL looking time (prediction: children will look less - totalLT - over time)

m <- lme(totalLT ~ trialnum, data = simulated_12session_data, random = ~ 1 | shortId) # LME model, grouping by participant
summary(m)

m <- lmer(totalLT ~ trialnum + (1 | shortId), data = simulated_12session_data) # LME model, grouping by participant
summary(m)

The key outcome variable is fracLessExpected (fractional looking time to the less-expected video of a pair) for all models. For now, leave out comparisons where this is not defined (need to handle these cases later)

BUT SEE alternate strategy proposed by Jesse: model fracLeft, and then ask whether the factor of interest (TargetLeft) is affected by the stimulus categories we care about.

• Create the appropriate nested random effects structure. There are two kinds of random effect we want to model: children and stimuli (children have within-condition comparisons, but stimuli do not!).

– What kind of slope terms are needed?

m <- lmer(sim_fracLessExpected  ~ 1 + (1|shortId) + (1|stimuli), data = simulated_12session_data) # LME model, grouping by participant * stimuli
summary(m)

• Key modeling question: Model Gravity, Support, and Control in 1 model, or split up?

We are less interested in differences among these than in characterizing the trajectory of each. So, ultimately expect to model separately? But for now, keep them together to write the ‘baseline’ model.

Within each (which I think we WILL want to keep together), there are (differently) nested sub-concepts, and then cross-cutting comparisons that test those concepts. As follows:

Concept > Event > comparison

NA > calibration > NA (need to characterize further)

Control > Salience > boring:interesting

Control > Same > A:B

Gravity > Ramp > down:up

Gravity > Toss > down:up (This IS assumed to be the same as for Ramp)

Support > Stay > “mostly-on:next-to”,“near:slightly-on”,“mostly-on:near”,“mostly-on:slightly-on”, “near:next-to”

Support > Fall > “mostly-on:next-to”,“near:slightly-on”,“mostly-on:near”,“mostly-on:slightly-on”, “near:next-to” (These levels ARE equivalent to those in Stay!)

For now, keep all together. This may become intractable!

• Expect that attention to fracLessExpected may vary for each event type, which are nested within concept types. E.g. Ramp and Fall events are both within the gravity concept. Ramp and Fall are not random (i.e. we don’t necessarily require ability to generalize to all possible ‘gravity tests’?), but they are related. That is, I think that conceptually the following is wrong (and the glmer package hates it), but not sure how to modify.

m <- lmer(sim_fracLessExpected  ~ event + concept + comparison + (1|shortId) + (1|stimuli), data = simulated_12session_data) # LME model, grouping by participant * stimuli
summary(m)

• Expect fractions to lessExpected to change as children get older, and, indepedently (???), as they have seen more sessions. Expect that both age and session terms interact with trial number (stuff gets more boring as you go along).

Conceptual point: How do we expect them to change? Linearly? Maybe not. In this kind of study, infants seem to have ‘sweet spots’ with some kind of stimuli, where they initially avoid a novel/complex stimulus, then tune into it. (But not in the other direction.) Quadratic slopes by concept?

m <- lmer(sim_fracLessExpected  ~ event + concept + comparison + session*trialnum + sim_age*trialnum + (1 | shortId) + (1|stimuli), data = simulated_12session_data) 
summary(m)

• …. Okay, pretend we settled on the base model! What we really want to know is whether kids have ‘interesting’ fracLessExpected values (ie different from 50% in either direction, with some constraints on the developmental trajectory), and if so, if their trajectories differ from one another for different concepts. In particular, we would like to estimate when the following concepts depart from chance, and if those estimates differ from one another. Ideally, we’d be able to say something about whether it’s the case that individual children ‘aquire concepts’ in a predictable order.

Control: Salience vs. Same (expect Same to stay at 50%!, by hypothesis/construction of stim there is no consistent difference!) Gravity vs. Support? Gravity Gravity vs. Same? (‘Same’ constitutes an expected baseline. Should it be a comparison for all other questions?) Gravity: Ramp vs. Toss Support: Across stay/fall Support: Stay vs. Fall (across comparisons, but expectations are opposite) Support: “mostly-on:next-to”,“near:slightly-on”,“mostly-on:near”,“mostly-on:slightly-on”, “near:next-to”

For this one, we have some hypothesis-driven predictions about order.

(THESE ARE HARD TO SPECIFY!)

In space: mostly-on slightly-on next-to near

Easiest: mostly-on:near Harder: Hardest: “near:next-to” “mostly-on:next-to”,“near:slightly-on”,“mostly-on:slightly-on”,

The problem

For the purposes of this preregistration, we assume that babies begin by not looking anywhere in particular, and then begin looking longer at surprising things (potentially at different points for different categories of things). This is a major simplifying assumption and may turn out to be flatly wrong in our dataset. See modeling section for details.

There is an uncomfortable fact that we expect that babies may actually look longer at either the less-surpising or more-surprising event, if they can tell the difference. That is, unequal attention in either direction is interpreted as ‘knowing the concept’, while equal attention (e.g. treating the comparison of ball tossing up/down the same as the comparison of oatmeal-stirring to oatmeal-stirring) is interpreted to mean no evidence for knowing that concept.

FORTUNATELY, this is constrained: a standard model is that we expect babies to initially look toward the more familiar thing (because the novel one is too confusing), and then shift toward looking at the surprising thing (which they can learn from/familiar one is too boring.) Qualitatively, it looks like this (from Hunter & Ames 1988):

The solution (for now)

Despite the above, in the realm of physical events specifically, most studies have shown infants to look longer at the surprising thing (though note that they have used single-item presentation rather than two-screen comparisons as we do here.) This suggests two possible solutions for modeling ‘learning curves’:

• Model linearly: Advantage